Storage system and remote copy recovery method

ABSTRACT

The storage system includes a first storage device, configured to be installed in a first site and providing a primary logical volume in which data received from a host computer is written; a second storage device, configured to be installed in a second site and providing a virtual logical volume, which has no structure for storing data received from the first storage system; and a third storage device, configured to be installed in a third site and providing a secondary logical volume in which data received from the second storage system is written. Data written in the primary logical volume of the first storage device are remote-copied to the secondary logical volume of the third storage device via the virtual logical volume of the second storage device.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation application of U.S. Ser. No.13/177,080, filed Jul. 6, 2011, which is a continuation application ofU.S. Ser. No. 12/194,625, filed Aug. 20, 2008 (now U.S. Pat. No.7,996,637), the entire disclosures of which are hereby incorporated byreference. This application relates to and claims priority from JapanesePatent Application No. JP 2008-173513, filed on Jul. 2, 2008, the entiredisclosure of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a storage system and a remote copyrecovery method, for example, is suitable for application to a 3DC (DataCenter) system.

2. Description of the Related Art

Conventionally, as countermeasures against natural calamities (disastersrecovery) in a storage system, a technique has been known that datawritten in a primary logical volume of a storage device operated by acertain site is copied to a secondary logical volume of a storage deviceinstalled in a site remote from the site for backup (hereinafter,referred to as “remote copy”).

In recent, a storage system has been proposed much, which is also calleda 3DC system, that data written in a primary logical volume of a firststorage device installed in a first site operated is transmitted to athird storage device installed in a third site via a second storagedevice installed in a second site, and, in the third storage device, thedata is copied to a secondary logical volume provided by the thirdstorage device.

For instance, in such 3DC system, a technique is disclosed in the PatentDocument 1 that the data as a transmitting object is transmitted to thethird storage device without an intermediate site (the second site)holding the data. According to such method, a logical volume with thesame capacity as the primary logical volume of the second storage deviceis not necessary to prepare, and thus there is an advantage to build the3DC system at a low cost.

-   [Patent Document 1] JP-A-2005-309550

However, in the 3DC system, when an obstacle occurs in a network forconnecting between, for example, the second and the third storagedevices, all of the data of remote copying object in the first storagedevice are transmitted to the third storage device via the secondstorage device after the corresponding network was recovered from theobstacle. With this, there is a problem in that it takes much time toreturn the operation of the 3DC system to a normal operation after thenetwork was recovered from the obstacle.

SUMMARY OF THE INVENTION

The present invention is directed to provide a storage system and aremote copy recovery method capable of reducing a recovery time from theobstacle of the remote copy.

The present invention is directed to solve such problems and to providea storage system including: a first storage device configured to beinstalled in a first site to provide a first logical volume; a secondstorage device configured to be installed in a second site; and a thirdstorage device configured to be installed in a third site to provide asecond logical volume, wherein data written in the first logical volumeof the first storage device are transmitted to the third storage devicevia the second storage device, the data being written in the samelocation as the first logical volume within the secondary logical volumein the third storage device; wherein, when transmission of the datastops among the first to the third storage devices, the respectivesecond storage device and the third storage device manage locations inthe secondary logical volume where the data held thereby are to bewritten; and wherein, when transmission of the data resumes among thefirst to the third storage devices, the locations in the secondarylogical volume managed by the respective second and the third storagedevices are aggregated, the data to be written in the respectiveaggregated location in the secondary logical volume being transmittedfrom the first storage device to the third storage device via the secondstorage device.

In addition, the present invention provides a remote copy recoverymethod in a storage system, in which data written in a first logicalvolume provided by a first storage device are transmitted to a thirdstorage device via a second storage device, the data being written inthe same location as the first logical volume within the secondarylogical volume in the third storage device, the method including: thefirst step of managing locations in the secondary logical volume wherethe data held thereby are to be written, in the respective secondstorage device and the third storage device, when transmission of thedata stops among the first to the third storage devices; and the secondstep of aggregating the locations in the secondary logical volumerespectively managed in the second and the third storage devices andtransmitting the data to be written in the respective aggregatedlocation in the secondary logical volume from the first storage deviceto the third storage device via the second storage device, whentransmission of the data resumes among the first to the third storagedevices.

According to a storage system and a remote copy recovery method, sincejust a difference between the first logical volume and the secondarylogical volume is copied at the time of transmitting of the data againamong the first to the third storage devices, a remote copy can berecovered into a normal state at a very short time, compared with aconventional method of copying all of the data in the first logicalvolume to the secondary logical volume.

According to the present invention, a storage system and a remote copyrecovery system capable of reducing a recovery time very shortly can berealized.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram to represent an entire configuration of astorage system according to the present embodiment;

FIG. 2 is a block diagram to represent configurations of the first tothe third storage devices;

FIG. 3 is a view to conceptually represent a configuration of the volumeinformation table;

FIG. 4 is a view to conceptually represent a configuration of the pairinformation table;

FIG. 5 is a view to conceptually represent a configuration of thejournal group setting information table;

FIG. 6 is a block diagram to be provided for description of the remotecopy process in the first storage device;

FIG. 7 is a block diagram to be provided for description of the remotecopy process in the second storage device;

FIG. 8 is a block diagram to be provided for description of the remotecopy process in the second storage device;

FIG. 9 is a block diagram to be provided for description of the remotecopy process in the third storage device;

FIG. 10 is a block diagram to be provided for description of the remotecopy process in the third storage device;

FIG. 11 is a block diagram to be provided for description of thesynchronization unlocking process in the first to the third storagedevices;

FIG. 12 is a block diagram to be provided for description of thesynchronization unlocking process in the second storage device;

FIG. 13 is a block diagram to be provided for description of thesynchronization unlocking process in the second storage device;

FIG. 14 is a conceptual view to be provided for description of thesynchronization unlocking process in the second storage device;

FIG. 15 is a block diagram to be provided for description of thesynchronization unlocking process in the third storage device;

FIG. 16 is a conceptual view to be provided for description of thesynchronization unlocking process in the third storage device;

FIG. 17 is a block diagram to be provided for description of thesynchronization unlocking process in the first storage device;

FIG. 18 is a block diagram to be provided for description of theresynchronization process performed in the second and the third storagedevices about resynchronization between the second and the third storagedevices;

FIG. 19 is a block diagram to be provided for description of theresynchronization process performed in the second storage device aboutresynchronization between the second and the third storage devices;

FIG. 20 is a conceptual view to be provided for description of theresynchronization process performed in the second storage device aboutresynchronization between the second and the third storage devices;

FIG. 21 is a block diagram to be provided for description of theresynchronization process performed in the third storage device aboutresynchronization between the second and the third storage devices;

FIG. 22 is a conceptual view to be provided for description of theresynchronization process performed in the third storage device aboutresynchronization between the second and the third storage devices;

FIG. 23 is a block diagram to be provided for description of theresynchronization process performed in the first and the second storagedevices about resynchronization between the first and the second storagedevices;

FIG. 24 is a block diagram to be provided for description of theresynchronization process performed in the first storage device aboutresynchronization between the first and the second storage devices;

FIG. 25 is a conceptual view to be provided for description of theresynchronization process performed in the first storage device aboutresynchronization between the first and the second storage devices;

FIG. 26 is a block diagram to be provided for description of theresynchronization process performed in the second storage device aboutresynchronization between the first and the second storage devices;

FIG. 27 is a conceptual view to be provided for description of theresynchronization process performed in the second storage device aboutresynchronization between the first and the second storage devices;

FIG. 28 is a time chart to be provided for description of theresynchronization process performed in the first to the third storagedevices about resynchronization among the first to the third storagedevices;

FIG. 29 is a flowchart to represent a process order of the commandreceiving process;

FIG. 30 is a flowchart to represent a process order of the writeprocess;

FIG. 31 is a flowchart to represent a process order of thesynchronization unlocking process;

FIG. 32 is a flowchart to represent a process order of the differencebit map transmitting process;

FIG. 33 is a flowchart to represent a process order of theresynchronization process;

FIG. 34 is a flowchart to represent a process order of the pair statechecking process;

FIG. 35 is a flowchart to represent a process order of the journalreading process;

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the accompanying drawings.

(1) Configuration of a Storage System According to the PresentEmbodiment

FIG. 1 shows a storage system 1 according to the present embodiment. Thestorage system 1 includes a first host computer to a third host computer3A to 3C and a first storage device to a third storage device 4A to 4Cinstalled in a first site to a third site 2A to 2C, respectively. Thefirst to the third sites 2A to 2C are, for example, computer facilitiesoperated by organizations such as a university, a company or the like,or data centers to operate a Web server on the Internet, an ASP(Application Service Provider), or the like. This storage system 1 isbuilt for a disaster discovery from an earthquake, a fire, a typhoon, aflood, the falling of a thunderbolt, a terror, or the like.

The first host computer 3A and the first storage device 4A are installedin the first site 2A of three sites. The second host computer 3B and thesecond storage device 4B are installed in the second site 2B, and thethird host computer 3C and the third storage device 4C are installed inthe third site 2C.

The first host computer 3A and the first storage device 4A installed inthe first site 2A are operated for a business, and the third hostcomputer 3C and the third storage device 4C installed in the third site2C are operated for a backup. In addition, the second host computer 3Band the second storage device 4B installed in the second site 2B areused for an intermediate upon transmission of data between the first andthe third storage devices 4A and 4C.

The first to third host computers 3A to 3C installed in the first to thethird sites 2A to 2C, respectively, are coupled communicably via anetwork 5. The network 5 is a WAN (Wide Area Network) configured of anIP (Internet Protocol) network, for example.

The first and the second storage devices 4A and 4B, and the second andthe third storage devices 4B and 4C are coupled via a network 6A and anetwork 6B, respectively. The networks 6A and 6B in this case employ,for example, an ATM (Asynchronous Transfer Mode) or a fiber channel,which has a high speed and a large capacity, in order to transmit copydata at the time of remote copy between the second and the third storagedevices 4B and 4C, described later.

The first to the third host computers 3A to 3C are computer deviceshaving hardware resources such as a CPU (Central Processing Unit) andmemories, respectively, and, are configured of a personal computer, aworkstation, a mainframe, or the like, for example.

Each of the first to the third storage devices 4A to 4C includes a diskarray unit 11 configured of a plurality of hard disk devices 10 and acontrol unit 12 to control input/output of data with respect to the harddisk devices 10, as shown in FIG. 2.

The hard disk devices 10 are configured of, for example, a disk of highprice such as an FC (Fiber Channel) or the like, or a disk of low pricesuch as a SATA (Serial AT Attachment), or the like. One or a pluralityof logical volumes VOL (hereinafter, referred to as “logical volume”)are created in storage zones provided by the one or a plurality of harddisk devices 10. Data are stored in such logical volumes VOL in a unitof a predetermined size of blocks (hereinafter, referred to as “logicalblock”).

A unique identifier (hereinafter, referred to as “logical volumenumber”) is given to each of the logical volumes VOL. In the presentembodiment, the input/output of the data is performed by designating anaddress which is a combination of the logical volume number and a uniquenumber (LBA: Logical Block Address) of the logical block given to eachof the logical block.

The control unit 12 includes a plurality of channel adaptors 20, aconnector 21, a share memory 22, a cache storage device 23 and aplurality of disk adaptors 24.

Each of the channel adaptors 20 is configured of a micro computer systemhaving a microprocessor, a memory and a communication interface and thelike, which serves as an communication interface among the first to thethird host computers 3A to 3C connected to a self-storage device, oramong the first to the third storage devices 4A to 4C coupled to theself-storage device via a network.

The connector 21 is connected to the channel adaptors 20, the sharememory 22, the cache memory 23 and the disk adaptors 24. Giving andreceiving of data or commands among the channel adaptors 20, the sharememory 22, the cache memory 23 and the disk adaptors 24 is performedover the connector 21. The connector 21 is configured of a switch or abus such as a super-speed cross bus switch to perform data transmissionby a high speed switching.

The share memory 22 is a memory shared by the channel adaptors 20 andthe disk adaptors 24. Information relating to configuration informationof the storage devices is stored in the share memory 22. As describedlater, a volume information table 30 (FIG. 3), a pair information table31 (FIG. 4) and a journal group setting information table 32 (FIG. 5)are stored in the share memory 22 as well.

The cache memory 23 is also a memory shared by the channel adaptors 20and the disk adaptors 24. The cache memory 23 is mainly used totemporarily memorize data input/output to/from the logical volumes VOL.

Each of the disk adaptors 24 is configured of a micro computer systemhaving a microprocessor or memories and serves as an interface toperform a protocol control at the time of communication of each of thehard disk devices 10. The disk adaptors 24 are coupled to thecorresponding hard disk devices 10 via, for example, a fiber channelcable and perform giving and receiving of data among the hard diskdevices 10 according to a fiber channel protocol.

(2) Flow of Remote Copy in the Present Storage System

The flow of remote copy in the present storage system 1 will now bedescribed using FIG. 1. Hereinafter, it is assumed that, a first logicalvolume VOL1 and a second logical volume VOL2 are installed in the firststorage device 4A and data from the first host computer are read orwritten from or in the first logical volume VOL1. That is to say, thefirst logical volume VOL1 becomes a primary logical volume in the 3DCsystem in the present storage system 1.

Additionally, it is assumed that, a virtual logical volume VVOL(hereinafter, referred to as “virtual volume”), which is set as a pairof the first logical volume VOL1 and has no substance, and a firstjournal volume JVOL1 storing a journal described later are installed inthe second the storage device 4B. A term “copy pair” means a pair of alogical volume VOL of a copy source and a logical volume VOL of a copydestination set to perform a data copy between the first and the secondsites 2A and 2B, or the second and the third sites 2B and 2C.

In addition, it is assumed that, in the third storage device 4C, thesecond journal volume JVOL2 set as a copy pair of the first journalvolume JVOL1, and a third logical volume VOLS which is a final backupvolume (secondary logical volume) of the first logical volume VOL1 areinstalled.

First, a journal will be described. The journal is update historyinformation of a certain logical volume created at the time of update ofdata stored in the logical volume, which is configured of the updatedata itself and information (hereinafter, referred to as “updateinformation”) such as an address within the logical volume where theupdate has been performed and a update time.

Data image in the logical volume of the copy source at a certain timepoint is copied to the logical volume of the copy destination(hereinafter, referred to as “initial copy”); thereafter, every timedata is written in the logical volume of the copy source, the journal iscreated, which is transmitted to the storage device of the copydestination.

The storage device of the copy destination reflects the transmittedjournal to the logical volume of the copy destination in an older order.With this, when the data image in the logical volume of the copy sourceis updated, the data image in the logical volume of the copy destinationcan be updated. “The reflection of the journal” means that write dataincluded in the journal is written in a designated address location bythe update information included in such journal in the logical volume ofthe copy destination.

A group of logical volumes, which is configured of the logical volumeand the journal volume storing the journal relating to the logicalvolume, is referred to as “a journal group.” In the present storagesystem, one journal group JG1 is constructed by a virtual volume VVOLand the first journal volume JVOL1 and another journal group JG2 isconstructed by the second journal volume JVOL2 and the third logicalvolume.

The flow of remote copy in the present storage system 1 is describedcontinuously. When receiving a write command and write data for thefirst logical volume VOL1 from first host computer 3A (the arrow in FIG.1), the first storage device 4A stores the write data in an addresslocation within the first logical volume VOL1, designated by the writecommand. Additionally, the first storage device 4A transmits such writedata to the second storage device 4B along with a write command to causethe virtual volume VVOL to be set as a recording destination (the arrowa1 in FIG. 1).

Once the second storage device 4B receives the write command and thewrite data sent from the first storage device 4A, it creates a journalbased on the write command and the write data, and then stores thecreated journal in the first journal volume JVOL1 (the arrow a2 in FIG.1). In addition, the second storage device 4B transmits the journalstored in the first journal volume JVOL1 to the third storage device 4C,asynchronously with transmission of the write command and the write datafrom the first storage device 4A (the arrow a3 in FIG. 1).

The third storage device 4C stores the journal transmitted from thesecond storage device 4B in the second journal volume JVOL2. The thirdstorage device 4C reads the journal stored in the second journal volumeJVOL2 and reflects the corresponding journal to the third logical volumeVOLS (the arrow a4 in FIG. 1)

Methods of remote copying the journal stored in the first journal volumeJVOL1 of the second storage device 4B to the second journal volume JVOL2of the third storage device 4C includes a method in which the secondstorage device 4B as the subject transmits the journal to the thirdstorage device 4C (hereinafter, referred to as “PUSH method”), and amethod in which the third storage device 4C as the subject reads out thejournal from the second storage device 4B (hereinafter, referred to as“PULL method”).

In the PULL method, a command for reading the journal (hereinafter,referred to as “journal read command”) is given to the second storagedevice 4B from the third storage device 4C. When the second storagedevice 4B receives the journal read command from the third storagedevice 4C, it reads the journal not-transmitted to the third storagedevice 4C from the first journal volume JVOL1 for transmission to thethird storage device 4C. Hereinafter, it is assumed that the PULL methodis employed as a method of transmitting the journal to the third storagedevice 4C from the second storage device 4B.

As means for performing the remote copy described above, the sharememory 22 of the first to third storage devices 4A to 4C stores thevolume information table 30 as shown in FIG. 3 and the pair informationtable 31 as shown in FIG. 4, and the share memory 22 of the second andthe third storage devices 4B and 4C stores the journal group settinginformation table 32 as shown in FIG. 5.

The volume information table 30 (FIG. 3) is a table by which the channeladaptors 20 of the first to the third storage devices 4A to 4C managethe logical volumes VOL set in the self-storage devices, which is, asshown in FIG. 3, configured of a logical volume number column 30A, avolume state column 30B and a pair number column 30C.

The logical volume numbers given to the respective logical volumes VOL(also include the virtual volume VVOL and the journal volumes JVOL1 andJVOL2) in the first to the third storage devices 4A to 4C are stored inthe logical volume number column 30A. The states of the correspondinglogical volumes VOL are stored in the volume state column 30B, and, inthe pair column 30C, when a corresponding logical volume VOL is set as acopy pair of another logical volume VOL, a unique identifier(hereinafter, referred to as “pair number”) given to the copy pair isstored. In addition, the pair number is the same as a pair number storedin a pair number column 31A of the pair information table 31 describedlater in FIG. 4.

Thus, an example of FIG. 3 shows that a state of the logical volume VOL(the virtual volume VVOL) having the logical volume number “1” is“normal,” and the logical volume VOL configures the copy pair having thepair number “1” together with another logical volume VOL (the firstlogical volume VOL1).

The pair information table 31 (FIG. 4) is a table for managing a copypair by the channel adopters 20 in the first storage device 4A to thethird storage device 4C, in case that the logical volume VOL registeredin the above volume information table 30 is set as such copy pair ofanother logical volume VOL set in the self-storage device or the otherstorage devices. The pair information table 31 is configured of a pairnumber column 31A, a pair state column 31B, an intra-device logicalvolume number column 31C, an other device number column 31D, an otherlogical volume number column 31E and a virtualized flag column 31F, asshown in FIG. 5.

A pair number given to the corresponding copy pair is stored in the pairnumber column 31A, and a state of the corresponding copy pair is storedin the pair state column 31B. The state of the copy pair includes“synchronization” that a update content in the logical volume VOL of thecopy source is synchronously or asynchronously reflected to the logicalvolume VOL of the copy destination, “obstacle” that the update contentin the logical volume VOL of the copy source is not reflected to thelogical volume VOL of the copy destination, and “synchronizing” that adifference between the logical volume VOL of the copy source and thelogical volume VOL of the copy destination is reflected to the logicalvolume VOL of the copy destination in order to transfer to the“synchronization” state from the “obstacle” state.

In the intra-device logical volume column 31C, logical volume numbers ofthe logical volumes VOL within the first to third storage devices 4A to4C, configuring the corresponding copy pair are stored. In the otherlogical volume number column 31E, logical volume numbers of logicalvolume VOL of the other side (hereinafter, referred to as “the otherside logical volume”) configuring the copy pair together with thelogical volumes VOL of which the logical volume numbers VOL is stored inthe intra-device logical volume number column 31C are stored. In theother device number column 31D, unique identifiers (hereinafter,referred to as “device number”) given to the storage devices (the firststorage device 4A to the third storage device 4C) having the other sidelogical volumes VOL are stored.

Flags to represent whether the other logical volumes VOL are virtualvolumes or not (hereinafter, referred to as “virtualization flag”) arestored in the virtualized flag 31F. Such virtualized flag is set to ONwhen the other logical volume VOL is a virtual volume and is set tooff(0) when the other logical volume VOL is not a virtual volume (thatis, a typical logical volume with a substance).

Therefore, it is known from an example of FIG. 4 that the copy pairhaving the pair number “1” is in a state (“synchronization”) that asynchronization has been made between the logical volume VOL of the copysource and the logical volume VOL of the copy destination, and,regarding to the other logical volume VOL, the logical volume number “1”set in the storage device having the device number “1” is a virtualvolume (the virtualization flag is “ON.”)

The journal group setting information table 32 (FIG. 5) is a table bywhich the channel adaptors 20 of the second and the third storagedevices 4B and 4C manage the journal groups JG1 and JG2 (FIG. 1) set inthe self-storage devices, and, as shown in FIG. 5, is configured of ajournal group number column 32A, a source volume column 32B, a journalvolume number column 32C, a path obstacle detection checking time column32D and a path disconnection detecting time column 32E.

A unique identifier (hereinafter, referred to as “journal group number”)given to each of the journal groups JG1 and JG2 set in the second or thethird storage device 4B or 4C is stored in the journal group numbercolumn 32A. A logical volume number of the journal volume (the first orthe second journal volume JVOL1 or JVOL2) belonging to the journalgroups JG1 and JG2 is stored in the journal volume column 32C, and alogical volume number of the logical volume VOL configuring the journalgroups JG1 and JG2 together with the journal volume is stored in thesource volume column 32B.

In addition, in the path obstacle detection checking time column 32D, atime is stored that it takes for the channel adaptors 20 to recognizeoccurrence of an obstacle after the obstacle occurs in paths (thenetworks 6A and 6B) for transmission/reception of the journal of thecorresponding journal groups JG1 and JG2 (hereinafter, the time isreferred to as “path obstacle detection checking time.”). In the presentstorage system 1, the time is a time until the channel adaptors 20recognize the occurrence of the obstacle in the networks 6A and 6B afterno receipt of a heart beat signal mutually exchanged between the secondand the third storage devices 4B and 4C. The time is set by a user andis stored in the path obstacle detection checking time column 32D as apath obstacle detection checking time. A time(year/month/day/hour/minute) when the channel adaptors 20 detect theobstacle of the path is stored in the path disconnection detecting timecolumn 32E.

Thus, an example of FIG. 5 shows that the journal group JG1 or JG2having the journal group number “1” is configured of the logical volumeVOL having the logical volume number “1” and the journal volume JVOL1 orJVOL2 having the logical volume number “3,” and the path obstacledetection checking time is set to “5 minutes.”

FIGS. 6 to 10 show flows of commands and data within the first to thethird storage devices 4A to 4C at the time of the remote copy describedabove. In FIGS. 6 to 10, a thin arrow represents a flow of command and athick arrow represents a flow of data.

In FIGS. 6 to 10, a command receipt processing program 40, a commandtransmission processing program 41, a journal read processing program43, a restore processing program 44 and a difference processing program45 are processing programs embedded in the channel adaptors 20, andread/write processing program 42 is a processing program embedded in thedisk adaptors 24.

In FIGS. 6 to 10, when the thin arrow having a certain processingprogram as a start point is drawn, it represents that a command isissued towards an endpoint of the arrow from the processing program,and, when the thick arrow having a processing program as a start pointor passing through the processing program is drawn, it represents thatdata are transmitted along the arrow based on the processing program.

As shown in FIG. 6, when a write command and write data are given to thefirst storage device 4A from the first host computer 3A, the channeladaptors 20 of the first storage device 4A to have received the writecommand and the write data save the write data in the cache memory 23(the arrow a10 in FIG. 6) as well as save the write command in the sharememory 22.

The write data saved in the cache memory 23 are read out from the cachememory 23 by the corresponding disk adaptors 24 and then are stored inan address location, designated by the write command, within the logicalvolume VOL1 designated by the write command saved in the share memory 22(the arrow all in FIG. 6).

The channel adaptors 20 to have received the write command and the writedata transmit a command (hereinafter, referred to as “data transmissioncommand”), which causes the write data related to the channel adaptor 20connected to the second storage device 4B to be transmitted to thesecond storage device 4B with reference to the volume information table30 (FIG. 3) and the pair information table 31 (FIG. 4) (the arrow a12 inFIG. 6).

The channel adaptor 20 to have received such data transmission commandreads the write data stored in the cache memory 23 with reference to thevolume information table 30 (FIG. 3) and the pair information table 31and transmits to the second storage device 4B the read write datatogether with a write command which designates the virtual volume VVOLas a write destination (the arrow a13 in FIG. 6).

On the other hand, the channel adaptor 20 of the second storage device4B to have received such write command and the write data stores thewrite command in the share memory 22 and stores the write data in thecache memory 23, as shown in FIG. 7. The channel adaptor 20 createsupdate information of the journal based on the write command for storingin the cache memory 23 (the arrow a20 in FIG. 7).

The channel adaptor 20 reports completion of writing of the write datato the first storage device 4A. The channel adaptor 20 of the firststorage device 4A receiving the report transmits a notification of thedata writing completion to the first host computer 3A via the channeladaptors 20 connected to the first host computer 3A.

In addition, the write data and the update information stored in thecache memory 23 of the second storage device 4B are read from the cachememory 23 by the disk adaptor 24 in charge of the first journal volumeJVOL1 and are stored in the first journal volume JVOL1 as a journal ofthe write data (the arrows a21 and a22 in FIG. 7).

Meanwhile, as shown in FIG. 9, the channel adaptor 20 connected to thesecond storage device 4B of the channel adaptors 20 in the third storagedevice 4C transmits a read journal command to the second storage device4B at a convenient timing depending on a situation of load thereof (thearrow a30 in FIG. 9).

The channel adaptor 20 of the second storage device 4B to have receivedthe read journal command, as shown in FIG. 8, examines whether journalsnot transmitted to the third storage device 4C are present in the firstjournal volume JVOL1. The channel adaptor 20 transmits a read commandfor such journals to the disk adaptor 24 in charge of the first journalvolume JVOL1 when the journals are present in the first journal volumeJVOL1.

The disk adaptor 24 to have received such read command reads theearliest journal in a creation order of the journals stored in the firstjournal volume JVOL1 from the first journal volume JVOL1 and stores itin the cache memory 23 (the arrow a31 and a32 in FIG. 8). The journalstored in the cache memory 23 is then read from the cache memory 23 bythe channel adaptor 20 connected to the third storage device 4C and istransmitted to the third storage device 4C (the arrow a33 in FIG. 8).

In the third storage device 4C, as shown in FIG. 9, the channel adaptor20 to have received such journal stores the corresponding journal in thecache memory 23 (the arrow a34 in FIG. 9). The journal stored in thecache memory 23 is then read from the cache memory 23 by the diskadaptor 24 in charge of the second journal volume JVOL2 and is stored inthe second journal volume JVOL2 as a journal (the arrows a35 and a36 inFIG. 9).

Thereafter, as shown in FIG. 10, a restore command is given to the diskadaptor 24 in charge of the second journal volume JVOL2 from the aspecific or an arbitrary channel adaptor 20 (the arrow a37 in FIG. 10).The disk adaptor 24 to have received the restore command reads thejournals stored in the second journal volume JVOL2 one by one to storethe update information in the journal in the cache memory 23 as well asstore the write data in the journal in a cache region 23A within thecache memory 23 (the arrow a38 in FIG. 10).

The write data stored in the cache region 23A is read by the diskadaptor 24 in charge of the third logical volume VOL3. The disk adaptor24 writes the associated write data in the corresponding addresslocation within the third logical volume VOL3 with reference to thecorresponding update information stored in the cache memory 23 (thearrow a39 in FIG. 10).

As described above, in the present storage system 1, the data written inthe first logical volume VOL1 is remote-copied to the third logicalvolume VOLS via the second storage device 4B.

(3) A Synchronization Unlocking Process and a Resynchronization ProcessAmong the First to the Third Storage Devices

A synchronization unlocking process performed in the second and thethird storage devices 4B and 4C, which stops the remote copy of thejournals between the second and the third storage devices 4B and 4C dueto occurrence of an obstacle in the network 6B connecting the second andthe third storage devices 4B and 4C or by indication of a user, and, aresynchronization process performed in the first to the third storagedevices 4A to 4C by manipulation of a user upon resumption of suchremote copy process, will now be described using FIGS. 11 to 28.

In FIGS. 11 to 28, a thin arrow represents a flow of command and a thickarrow represents a flow of data. In FIGS. 11 to 28, a command receiptprocessing program 40, a command transmission processing program 41, ajournal read processing program 43, a restore processing program 44, adifference processing program 45 and a pair state check processingprogram 46 are processing programs embedded in the channel adaptors 20,and read/write processing program 42 is a processing program embedded inthe disk adaptors 24.

In addition, in FIGS. 11 to 28 except for FIGS. 14, 16, 20, 22, 25 and27, when the thin arrow having a certain processing program as a startpoint is drawn, it represents that a command is issued towards anendpoint of the arrow from the processing program, and, when the thickarrow having a processing program as a start point or passing throughthe processing program is drawn, it represents that data are transmittedalong the arrow based on the processing program.

(3-1) Synchronization Unlocking Process

First, the synchronization unlocking process will be described. When thesecond and the third storage devices 4B and 4C detect occurrence of anobstacle in the network 6B or an indication for stopping the remote copybetween the second and the third storage devices 4B and 4C is given froma user, each of them manages data, which are held by the self-storagedevice, not reflected to the third logical volume VOLS, as shown in FIG.11.

In detail, the second and the third storage devices 4B and 4C manageaddress locations within the third logical volume VOL3 reflecting therespective journals stored in the first or the second journal volumeJVOL1 or JVOL2 in the self-storage device, using a secondary inter-P/Idifference bit map 51B or a secondary inter-I/R difference bit map 51B.

The first storage device 4A manages an address location, which isupdated after stopping of the remote copy between the second and thethird storage devices 4B and 4C, within the first logical volume VOL1,using a primary inter-P/I difference bit map 51A.

FIG. 12 shows a flow of the synchronization unlocking process performedin the second storage device 4B when the second storage device 4Bdetects an obstacle occurred in the network 6B.

The channel adaptor 20 connected to the third storage device 4C, of thechannel adaptors 20 of the second storage device 4B, transmits apredetermined signal for check to the third storage device 4C based onthe pair state check processing program 46, and, always checks a stateof the network 6B connecting the second and the third storage devices 4Band 4C based on a response or not of the third storage device 4C to thesignal for check (the arrow a40 in FIG. 12).

When the channel adaptor 20 transmits the signal for check to the thirdstorage device 4C and then does not receive a response from the thirdstorage device 4C even after a time elapsed exceeding the correspondingpath obstacle detection checking time set in the journal group settinginformation table 32 (FIG. 5), it transmits a synchronization unlockingcommand to the channel adaptor 20 connected to the first storage device4A (the arrow a41 in FIG. 12).

The channel adaptor 20 to have received such synchronization unlockingcommand first changes into an “obstacle” the pair state stored in thepair state column 31B of the entry corresponding to the copy pairconfigured of the first and the second journal volumes JVOL1 and JVOL2,of the entries in the pair information table 31, as shown in FIG. 14.

Subsequently, the channel adaptor 20 issues a read command which readsthe update information in one journal of the journals stored in thefirst journal volume JVOL1 at such time point so as to be stored in thecache memory 23, to the disk adaptor 24 in charge of the first journalvolume JVOL1 of the synchronization unlocking object (the arrow a42 inFIG. 12).

The disk adaptor 24 to have received such read command holds the writedata and the update information thereof not stored in the first journalvolume JVOL1 yet of the write data and the update information thereofheld in the cache memory 23, to the first journal volume JVOL1 as ajournal. The disk adaptor 24 then reads update information in onejournal stored in the first journal volume JVOL1 and stores the readupdate information in the cache memory 23 (the arrow a43 in FIG. 12).

Successively, the channel adaptor 20 to have received thesynchronization unlocking command creates a difference bit map(hereinafter, referred to as “primary inter-I/R difference bit map”) 50Athat a bit is installed by mapping each block of the first logicalvolume VOL1 provided by the first host computer 3A, on the share memory22. All of bits on the primary inter-I/R difference bit map 50A are setto off (“0”) at the beginning.

The channel adaptor 20 to have received the synchronization unlockingcommand determines an address location in on the third logical volumeVOL3 to which the corresponding journal is to be reflected, based on theread update information from the cache memory 23, and sets a bitcorresponding to the address location on the primary inter-P/Idifference bit map 50A to ON (the arrow a44 in FIG. 12).

Thereafter, the channel adaptor 20 performs the same process for all ofthe journals stored in the first journal volume JVOL1. With this, anaddress location in the third logical volume VOL3 to which the journalsheld in the first journal volume JVOL1 are to be reflected, isregistered in the primary inter-I/R difference bit map 50A, at the timeof detection of the obstacle of the network 6B connecting the second andthe third storage devices 4B and 4C.

In the meantime, FIG. 13 shows a detailed process content of the secondstorage device 4B, for example, in a case of the second host computer 3Bgiving a synchronization unlocking command (the arrow a50 in FIG. 13)for unlocking a synchronization between the first and the second journalvolumes JVOL1 and JVOL2 to the second storage device 4B.

In this case, the channel adaptor 20 of the second storage device 4B tohave received the synchronization unlocking command transmits it to thechannel adaptor 20 connected to the third storage device 4C (the arrowa51 in FIG. 13). The channel adaptor 20 connected to the third storagedevice 4C transmits the received synchronization unlocking command tothe third storage device 4C (the arrow a52 in FIG. 13).

The channel adaptor 20 connected to the first storage device 4A performsthe same process as described for the arrows a42 and a44 in FIG. 12. Asa result, similar to FIG. 12, in this case, too, an address location inthe third logical volume VOL3 to which the journal held in the cachememory 23 or the first journal volume JVOL1 is to be reflected isregistered in the primary inter-I/R difference bit map 50A, at the timepoint when such synchronization unlocking command is given to the secondstorage device 4B.

FIG. 15 shows a flow of a process of the third storage device 4C whenthe third storage device 4C detects an obstacle occurred in the network6B connecting the second and the third storage devices 4B and 4 c.

When the channel adaptor 20 connected to the second storage device 4B ofthe channel adaptors 20 of the third storage device 4C transmits a readjournal command to the second storage device 4B according to a requestfrom itself or different channel adaptors 20 (the arrow 60 a in FIG.15), it continues to measure a time until a response is received afterthe corresponding read journal command was transmitted.

When the channel adaptor 20 transmits the read journal command to thesecond storage device 4B and then does not receive a response from thesecond storage device 4B even after a time elapsed exceeding thecorresponding path obstacle detection checking time set in the journalgroup setting information table 32 (FIG. 5), it transmits asynchronization unlocking command to the channel adaptor 20 connected toa specific or an arbitrary channel adaptor 20 (the arrow a61 in FIG.15).

The channel adaptor 20 to have received such synchronization unlockingcommand first updates into an “obstacle” the pair state stored in thepair state column 31B of the entry corresponding to the copy pairconfigured of the first and the second journal volumes JVOL1 and JVOL2,of the entries in the pair information table 31, as shown in FIG. 16.

Subsequently, the channel adaptor 20 to have received thesynchronization unlocking command issues a read command which reads theupdate information in one journal of the journals stored in the secondjournal volume JVOL2 at such time point, to the disk adaptor 24 incharge of the second journal volume JVOL2 of the synchronizationunlocking object (the arrow a62 in FIG. 15).

The disk adaptor 24 to have received such read command stores the writedata and the update information thereof not stored in the second journalvolume JVOL2 yet of the write data and the update information thereofheld in the cache memory 23, in the second journal volume JVOL2 as ajournal. The disk adaptor 24 then reads update information in onejournal stored in the second journal volume JVOL2 from the secondjournal volume JVOL2 and stores the read update information in the cachememory 23 (the arrow a63 in FIG. 15).

Successively, the channel adaptor 20 to have received thesynchronization unlocking command creates a difference bit map(hereinafter, referred to as “secondary inter-I/R difference bit map”)50B that a bit is installed by mapping each block of the first logicalvolume VOL1, on the share memory 22. All of bits on the secondaryinter-I/R difference bit map 50B are set to off (“0”) at the beginning.

The channel adaptor 20 to have received the synchronization unlockingcommand determines an address location in on the third logical volumeVOL3 to which the corresponding journal is to be reflected, based on theread update information from the cache memory 23, and sets a bitcorresponding to the address location on the secondary inter-I/Rdifference bit map 50B to ON (the arrow a64 in FIG. 15).

Thereafter, the channel adaptor 20 performs the same process for all ofthe journals stored in the second journal volume JVOL2. With this, anaddress location in the third logical volume VOL3 to which the journalsheld in the cache memory 23 or the second journal volume JVOL2 are to bereflected, is registered in the secondary inter-I/R difference bit map50B, at the time of detection of the obstacle of the network 6Bconnecting the second and the third storage devices 4B and 4C.

Also, when the synchronization unlocking command is given to the secondstorage device 4B via the first storage device 4A from the first hostcomputer 3A, the third storage device 4C performs the same process. Inthis case, as described in FIG. 13, the synchronization unlockingcommand is transmitted to the third storage device 4C from the secondstorage device 4B. The channel adaptor 20 of the third storage device 4Cto have received the synchronization unlocking command transmits thesynchronization unlocking command to a specific or an arbitrary channeladaptor 20.

The specific or the arbitrary channel adaptor 20 to have received thesynchronization unlocking command performs the synchronization unlockingprocess as described above. With this, an address location in the thirdlogical volume VOL3 to which the journals held in the second journalvolume JVOL2 are to be reflected, is registered in the secondaryinter-I/R difference bit map 50B which is created in the third storagedevice 4C, at the time of transmission of the synchronization unlockingcommand from the second storage device 4B.

FIG. 17 shows a detailed process content of the first storage device 4Awhen the remote copy ceases between the second and the third storagedevices 4B and 4C due to an obstacle in the network 6B or a user'sindication.

As described in FIG. 6, when the first storage device 4A receives thewrite command and the write data from the first host computer 3A, ittransmits the write command to the second storage device 4B. The secondstorage device 4B reports occurrence of an obstacle to the first storagedevice 4A when it stops the remote copy with the third storage device4C.

The channel adaptor 20 of the first storage device 4A to have receivedsuch report of the obstacle occurrence changes into an “obstacle” thepair state stored in the pair state column 31B of the entrycorresponding to the copy pair configured of the first logical volumeVOL1 and the virtual volume VVOL of the entries in the pair informationtable 31 (FIG. 4) stored in the share memory 22.

When the channel adaptor 20 connected to the first host computer 3Areceives the write command which designates the first logical volumeVOL1 as a write destination of data and write data from the first hostcomputer 3A, it stores the write data in the cache memory 23 (the arrowa70 in FIG. 17) and immediately issues a write completion notificationto the first host computer 3A (the arrow a71 in FIG. 17).

Thereafter, the channel adaptor 20 sets to on a bit corresponding to ablock in the first logical volume VOL1 designated by the write commandreceived at that time, of the bits on a difference bit map (hereinafter,referred to as “primary inter-P/I difference bit map”) 51A stored in theshare memory 22 (the arrow a73 in FIG. 17). With this, locations in thefirst logical volume VOL1 updated after stopping of the remote copybetween the second storage device 4B and the third storage device 4C aremanaged by the primary inter-P/I difference bit map 51A.

In addition, the write data stored in the cache memory 23 are read fromthe cache memory 23 by the disk adaptor 24 in charge of the firstlogical volume VOL1 and are stored in a designated location in the firstlogical volume VOL1 by the associated write command (the arrow a74 inFIG. 17).

(3-2) Resynchronization Process

A resynchronization process will now be described. In the presentstorage system 1, after the network 6B between the second storage device4B and the third storage device 4C recovers and when the remote copyresumes between the first and the third logical volumes VOL1 and VOLS, auser manipulates the second host computer 3B to transmit aresynchronization command for resynchronization between the second andthe third sites 2B and 2C, to the second storage device 4B from thesecond host computer 3B, as shown in FIG. 28.

The second storage device 4B to have received the resynchronizationcommand transmits a difference bit map transmitting command fortransmission of the secondary inter-I/R difference bit map 50B to thethird storage device 4C, as shown in FIGS. 28 and 18 (SP100 in FIG. 28).The third storage device 4C to have received the difference bit maptransmitting command transmits the secondary inter-I/R difference bitmap 50B held in the share memory 22 at that time to the second storagedevice 4B (SP101 in FIG. 28). In addition, the second storage device 4Bto have received the secondary inter-I/R difference bit map 50B mergesthe secondary inter-I/R difference bit map 50B into the primaryinter-P/I difference bit map 51A held in the share memory 22 at thattime (SP102 in FIG. 28).

In each of the second and the third storage devices 4B and 4C, the pairstate of the copy pair configured of the first and the second journalvolume JVOL1 and JVOL2 returns to “synchronization” and thus the readjournal command issues again from the third storage device 4C to thesecond storage device 4B (SP103 in FIG. 28).

Additionally, the user manipulates the first host computer 3A totransmit a resynchronization command for resynchronization between thefirst and second sites 2A and 2B, to the first storage device 4A fromthe first host computer 3A, as shown in FIG. 28.

The first storage device 4A to have received the resynchronizationcommand transmits a difference bit map transmitting command fortransmission of the secondary inter-P/I difference bit map 51B to thesecond storage device 4B, as shown in FIGS. 28 and 23 (SP104 in FIG.28). The second storage device 4B to have received the difference bitmap transmitting command transmits the secondary inter-P/I differencebit map 51B held in the share memory 22 at that time to the firststorage device 4A (SP105 in FIG. 28). In addition, the first storagedevice 4A to have received the secondary inter-P/I difference bit map51B merges the secondary inter-P/I difference bit map 51B into theprimary inter-P/I difference bit map 51A held in the share memory 22 atthat time (SP106 in FIG. 28).

In each of the first and the second storage devices 4A and 4B, the pairstate of the copy pair configured of the first logical volume VOL1 andthe virtual volume VVOL returns to “synchronization,” and thus what iscalled a difference copy is performed that the data stored in each blockof which the corresponding bit is set to ON in the primary inter-P/Idifference bit map 51A, of the blocks in the first logical volume VOL1,are remote-copied between the first storage device 4A and the secondstorage device 4B (SP107 in FIG. 28).

When the difference copy is completed, the remote copy described inFIGS. 6 and 7 resumes between the first storage device 4A and the secondstorage device 4B.

Herein, FIG. 19 shows a content of a resynchronization process performedin the second storage device 4B when the resynchronization command isgiven to the second storage device 4B from the second host computer 3B,of a series of processes described in FIG. 28.

In this case, the channel adaptor 20 of the second storage device 4B tohave received the resynchronization command (the arrow a81 in FIG. 19)transmits the resynchronization command to the channel adaptor 20connected to the third storage device 4C (the arrow a82 in FIG. 19).

In addition, the channel adaptor 20 connected to the third storagedevice 4C to have received the resynchronization command transmits atransmission command of a difference bit map (hereinafter, referred toas “difference bit map transmitting command”) to the third storagedevice 4C (the arrow a83 in FIG. 19).

The channel adaptor 20 of the third storage device 4C to have receivedthe difference bit map transmitting command, as shown in FIG. 21, readsthe secondary inter-I/R difference bit map 50B from the share memory 22for transmission to the second storage device 4B (the arrow a84 in FIG.21). Such channel adaptor 20, as shown in FIG. 22, updates the pairstate stored in the pair state column 31B of the entry corresponding tothe copy pair configured of the first and the second journal volumeJVOL1 and JVOL2 of the entries in the pair information table 31 storedin the share memory 22, from the “obstacle” to a “synchronizing”.

The channel adaptor 20 of the second storage device 4B to have receivedsuch secondary inter-I/R difference bit map 50B merges the correspondingsecondary inter-I/R difference bit map 50B into the primary inter-I/Rdifference bit map 50A prepared in advance on the share memory 22, asshown in FIG. 21. In addition, the channel adaptor 20 then merges theprimary inter-I/R difference bit map 50A into the secondary inter-P/Idifference bit map 51B held in the share memory 22 (the arrow a85 inFIG. 21).

In detail, such channel adaptor 20 sets each bit on the primaryinter-I/R difference bit map 50A, corresponding to each bit set to ON onthe secondary inter-I/R difference bit map 50B, to ON. Thereafter, thechannel adaptor 20 sets each bit on the secondary inter-P/I differencebit map 51B, corresponding to each bit set to On on the primaryinter-I/R difference bit map 50A, to ON.

Subsequently, the channel adaptor 20 connected to the third storagedevice 4C, as shown in FIG. 20, updates the pair state stored in thepair state column 31B of the entry corresponding to the copy pairconfigured of the first and the second journal volume JVOL1 and JVOL2 ofthe entries in the pair information table 31 stored in the share memory22, from the “obstacle” to an “synchronizing”.

FIG. 24 shows a content of a resynchronization process performed in thefirst storage device 4A when the resynchronization command is given tothe first storage device 4A from the first host computer 3A, of a seriesof processes described in FIG. 28.

In this case, the channel adaptor 20 of the first storage device 4A tohave received the resynchronization command (the arrow a91 in FIG. 24)transmits the resynchronization command to the channel adaptor 20connected to the second storage device 4B (the arrow a92 in FIG. 24).

In addition, the channel adaptor 20 connected to the second storagedevice 4B to have received the resynchronization command transmits atransmission command of a difference bit map (hereinafter, referred toas “difference bit map transmitting command”) to the second storagedevice 4B (the arrow a93 in FIG. 24).

The channel adaptor 20 of the second storage device 4B to have receivedthe difference bit map transmitting command, as shown in FIG. 26, readsthe secondary inter-P/I difference bit map 51B from the share memory 22for transmission to the first storage device 4A (the arrow a94 in FIG.26). Such channel adaptor 20, as shown in FIG. 27, updates the pairstate stored in the pair state column 31B of the entry corresponding tothe copy pair configured of the first logical volume VOL1 and thevirtual volume VVOL of the entries in the pair information table 31stored in the share memory 22, from the “obstacle” to an“synchronizing”.

The channel adaptor 20 of the first storage device 4A to have receivedsuch secondary inter-P/I difference bit map 51B merges the correspondingsecondary inter-P/I difference bit map 51B into the primary inter-P/Idifference bit map 51A, as shown in FIG. 24 (the arrow a95 in FIG. 24).In detail, such channel adaptor 20 sets each bit on the primaryinter-P/I difference bit map 51A, corresponding to each bit set to ON onthe secondary inter-P/I difference bit map 51B, to ON.

Subsequently, the channel adaptor 20 connected to the third storagedevice 4C, as shown in FIG. 25, updates the pair state stored in thepair state column 31B of the entry corresponding to the copy pairconfigured of the first logical volume VOL1 and the virtual volume VVOLof the entries in the pair information table 31 stored in the sharememory 22, from the “obstacle” to an “synchronizing”.

(4) Process of the Channel Adaptor

A detailed process content performed by the channel adaptors 20 of thefirst to the third storage devices 4A to 4C to have received commandstransmitted from the first to the third host computers 3A to 3Cinstalled in the same site or the first to the third storage devices 4Ato 4C connected to self-storage devices, will be described.

(4-1) Command Receiving Process

FIG. 29 shows a process content of a command receiving process performedby the channel adaptors 20 to have received the commands transmittedfrom the first to the third host computers 3A to 3C installed in thesame site or the first to the third storage devices 4A to 4C connectedto self-storage devices.

When the channel adaptor 20 receives such command, it begins the commandreceiving process shown in FIG. 29 to determine whether the command is awrite command or not (SP1). If obtaining an affirmative result from thedetermination, the channel adaptor 20 performs a write process accordingto the command (write command) (SP2), and then finishes the commandreceiving process. The write command is receivable by the first and thesecond storage devices 4A and 4B but is not receivable by the thirdstorage device 4C.

If obtaining a negative result from the determination at the step SP1,the channel adaptor 20 determines whether the command is asynchronization unlocking command or not (SP3). If obtaining anaffirmative result from the determination, the channel adaptor 20performs the synchronization unlocking process for unlocking asynchronization of a copy pair designated by the synchronizationunlocking command (SP4), and then finishes the command receivingprocess. The synchronization unlocking command is a receivable commandby the first to the third storage devices 4A to 4C.

If obtaining a negative result from the determination at the step SP3,the channel adaptor 20 determines whether the command is a differencebit map transmitting command (SP5). If obtaining an affirmative resultfrom the determination, the channel adaptor 20 performs a difference bitmap transmitting process for transmitting the secondary inter-P/Idifference bit map 51B or the secondary inter-I/R difference bit map 50Bstored in the share memory 22 in the self-storage device to atransmitting source of the corresponding command (SP6), and thenfinishes the command receiving process. The difference bit maptransmitting command is receivable by the second and the third storagedevices 4B and 4C but is not receivable by the first storage device 4A.

On the other hand, if obtaining a negative result from the determinationat the step SP5, the channel adaptor 20 determines whether the commandis a resynchronization command or not (SP7). If obtaining an affirmativeresult from the determination, the channel adaptor 20 performs aresynchronization process for resynchronization between the first tothird storage devices 4A to 4C designated by the command (theresynchronization command) (SP8), and then finishes the commandreceiving process. The resynchronization command is receivable by any ofthe first to the third storage devices 4A to 4C.

If obtaining a negative result from the determination at the step SP7,the channel adaptor 20 performs a process according to the command(SP9), and then finishes the command receiving process.

(4-2) Write Process

FIG. 30 shows a detailed process content of a write process performed atthe step SP2 of the command receiving process described above. In thepresent embodiment, this write process is performed in a channel adaptor20 connected to the first host computer 3A of the channel adaptors 20 ofthe first storage device 4A or in a channel adaptor 20 connected to thefirst storage device 4A of the channel adaptors 20 of the second storagedevice 4B.

The channel adaptor 20 begins the write process upon going to the stepSP2 of the command receiving process described in FIG. 29. The channeladaptor 20 first determines whether a state of a logical volume ofwriting destination of the write data (hereinafter, the logical volumeis the first logical volume VOL1 or the virtual volume VVOL, which isreferred to as “data writing destination logical volume.”) designated bythe write command is normal or not (SP10), referring to the volumeinformation table 30 (FIG. 3) stored in the share memory 22.

If obtaining a negative result from the determination, the channeladaptor 20 reports the abnormal state to a transmitting source of thewrite command (the first host computer 3A or the first storage device4A)(SP11) and then finishes the write process to go back to the commandreceiving process in FIG. 29.

As compared therewith, if obtaining an affirmative result from thedetermination at the step SP10, the channel adaptor 20 stores the writedata together with the write command transmitted from the transmittingsource of the command in the cache memory 23 (SP12).

Successively, the channel adaptor 20 determines whether the data writingdestination logical volume designated by the write command is set as acopy pair of another logical volume or not, referring to the pairinformation table 31 (FIG. 4) stored in the share memory 22 (SP13).

If obtaining an affirmative result from the determination, the channeladaptor 20 reports completion of writing process of the write data tothe transmitting source of the write command (SP20) and then finishesthe write process to go back to the command receiving process in FIG.29. In addition, the write data stored in the cache memory 23 at thistime are read by the corresponding disk adaptor 24 and are written in anaddress location designated by the above-described write command of alogical volume designated by the corresponding write command.

If obtaining a negative result from the determination at the step SP13,the channel adaptor 20 determines whether a pair state of the copy pairconfigured of the data writing destination logical volume is“synchronization” or “synchronizing”, based on the pair informationtable 31 (SP14).

A negative result in the determination at this step means the pair stateof the copy pair is “obstacle.” However, since the first storage device4A does not transmit the write command to the second storage device 4Bafter the pair state of the copy pair configured of the first logicalvolume VOL1 and the virtual volume VVOL becomes “obstacle,” only thechannel adaptors 20 of the first storage device 4A obtains such negativeresult in the determination at this step.

The channel adaptor 20 sets a bit corresponding to a block of the datawriting destination designated by the write command received at thattime, of the bits on the primary inter-P/I difference bit map 51A storedin the share memory 22, to ON (SP21). The channel adaptor 20 reportscompletion of the writing process of data to the transmitting source ofthe write command (herein, the first host computer 3A) (SP22), and thenfinishes the write process to go back to the command receiving processin FIG. 29.

On the other hand, if the channel adaptor 20 obtains an affirmativeresult in the determination at the step SP14, it determines whether thedata writing destination logical volume belongs to any one of thejournal groups set in the self-storage device (SP16).

Herein, if the channel adaptor 20 belongs to the first storage device4A, the channel adaptor 20 obtains a negative result in thedetermination at the step SP15. In this case, the channel adaptor 20transmits the write data transmitted from the first host computer 3A tothe second storage device 4B (SP17), and then determines whether thetransmission of the write data is successful or not, based on a responseor not from the second storage device 4B (SP18).

If the channel adaptor 20 obtains an affirmative result in thisdetermination, it reports completion of the writing process of data tothe transmitting source of the write command (herein, the first hostcomputer 3A) (SP22), and then finishes the write process to go back tothe command receiving process in FIG. 29.

On the contrary, if the channel adaptor 20 obtains a negative result inthe determination at the step SP17, it changes the pair state of thecopy pair stored in the pair state column 31B (FIG. 4) of the entrycorresponding to the pair information table 31 into “obstacle.” (SP18)

The channel adaptor 20 sets a bit corresponding to a block designated bythe write command in a logical volume (the first logical volume VOL1)designated by the write command, of the bits on the primary inter-P/Idifference bit map 51A stored in the share memory 22, to ON (SP19).

The channel adaptor 20 reports completion of the writing process of datato the transmitting source of the write command (herein, the first hostcomputer 3A) (SP22), and then finishes the write process to go back tothe command receiving process in FIG. 29.

If the channel adaptor 20 belongs to the second storage device 4B, thecorresponding channel adaptor 20 obtains an affirmative result in thedetermination at the step S16. At this time, the channel adaptor 20creates a update history about the write data stored in the cache memory23 at the step SP12 and then stores it in the cache memory 23 (SP20).The update information created at this time is written in the firstjournal volume JVOL1 as a journal by the corresponding disk adaptor 24,together with the corresponding write data written in the cache memory23 at the step SP12.

(4-3) Synchronization Unlocking Process

FIG. 31 shows a detailed process content of the synchronizationunlocking process performed at the step SP4 of the above-describedcommand receiving process. In this embodiment, this synchronizationunlocking process is performed in a channel adaptor 20 connected to thefirst storage device 4A of the channel adaptors 20 of the second storagedevice 4B, or in a channel adaptor 20 connected to the second storagedevice 4B of the channel adaptors 20 of the third storage device 4C.

Such channel adaptor 20 begins the synchronization unlocking processupon going to the step SP4 of the command receiving process described inFIG. 29, and then changes a pair state stored in the pair state column31B of the entry corresponding to the copy pair of the synchronizationunlocking object designated by the synchronization unlocking command, ofthe entries of the pair information table 31 (FIG. 4), into “obstacle.”(SP30).

Subsequently, if journals (write data and update information thereof)not stored in the first and the second journal volumes JVOL1 and JVOL2are present on the cache memory 23, the channel adaptor 20 controls thecorresponding disk adaptor 24 to store the corresponding journal in thefirst journal volume JVOL1 or the second journal volume JVOL2 (SP31).

Thereafter, the channel adaptor 20 determines whether journals nottransmitted to the third storage device 4C or not reflected to the thirdlogical volume VOL3, in the first journal volume JVOL1 or in the secondjournal volume JVOL2 in the self-storage device, are present or not(SP32).

If obtaining a negative result in this determination, the channeladaptor 20 controls the corresponding disk adaptor 24 to read updateinformation in one journal of the journals stored in the first or thesecond journal volumes JVOL1 or JVOL2 from the cache memory 23 (SP33).

The channel adaptor 20 sets a bit corresponding to a block in the thirdlogical volume VOL3 to which the journal, from which the updateinformation is read at the step SP33, is reflected, of the bits on thesecondary inter-P/I difference bit map 51B or on the secondary inter-I/Rdifference bit map 50B stored in the share memory 22, based on theupdate information read by the disk adaptor 24 at this time, to ON(SP34).

Moreover, the channel adaptor 20 liberates the storage zone storing thejournal in the first or the second journal volume JVOL1 or JVOL2 (SP35).This eliminates the journal from the first or the second journal volumeJVOL1 or JVOL2.

Thereafter, the channel adaptor 20 repeats the processes of the stepsSP32 to SP35 until obtaining a negative result at the step SP32. Withthis, each bit on the secondary inter-P/I difference bit map 51B or onthe secondary inter-I/R difference bit map 50B corresponding to eachaddress location in the third logical volume VOLS to be updated by thejournals stored in the first or the second journal volume JVOL1 or JVOL2at that time, is set to ON one by one.

At last, if the channel adaptor 20 obtains a negative result in thedetermination at the step SP32 by performing the above-describedprocesses for all of the journals stored in the first or the secondjournal volume JVOL1 or JVOL2, it finishes the synchronization unlockingprocess to go back to the command receiving process in FIG. 29.

(4-4) Difference Bit Map Transmitting Process

FIG. 32 shows a detailed process content of the difference bit maptransmitting process performed at the step SP6 of the above-describedcommand receiving process. In the present embodiment, this differencebit map transmitting process is performed in a channel adaptor 20connected to the first storage device 4A of the channel adaptors 20 ofthe second storage device 4B or in a channel adaptor 20 connected to thesecond storage device 4B of the channel adaptors 20 of the third storagedevice 4C.

Such channel adaptor 20 begins the difference bit map transmittingprocess shown in FIG. 32 upon going to the step SP6 of the commandreceiving process described in FIG. 29. The channel adaptor 20 reads thesecondary inter-P/I difference bit map 51B or the secondary inter-I/Rdifference bit map 50B from the share memory 22 for transmission to thetransmitting source of the difference bit map transmitting command (thefirst or the second storage device 4A or 4B) (SP40).

Then, the channel adaptor 20 changes a pair state (if the channeladaptor 20 belongs to the second storage device 4B) stored in the pairstate column 31B of an entry corresponding to the copy pair configuredof the first logical volume VOL1 and the virtual volume VVOL of theentries of the pair information table 31 (FIG. 4) stored in the sharememory 22, or a pair state (if the channel adaptor 20 belongs to thethird storage device 4C) stored in the pair state column 31B of an entrycorresponding to the copy pair configured of the first and the secondjournal volumes JVOL1 and JVOL2 of the entries of the pair informationtable 31 (FIG. 4), into “synchronization” (SP41).

The channel adaptor 20 finishes the difference bit map transmittingprocess and then goes back to the command receiving process described inFIG. 29.

(4-5) Resynchronization Process

FIG. 33 shows a detailed process content of the resynchronizationprocess performed at the step SP8 of the above-described commandreceiving process. In the present embodiment, this resynchronizationprocess is performed in a channel adaptor 20 connected to the secondstorage device 4B of the channel adaptors 20 of the first storage device4A or in a channel adaptor 20 connected to the third storage device 4Cof the channel adaptors 20 of the second storage device 4B.

Such channel adaptor 20 begins the resynchronization process upon goingto the step SP8 of the command receiving process described in FIG. 29.The channel adaptor 20 transmits a pair resynchronizing request forrequesting a resynchronization of the copy pair configured of the firstlogical volume VOL1 and the virtual volume VVOL (if the channel adaptor20 belongs to the first storage device 4A) or of the copy pairconfigured of the first and the second journal volumes JVOL1 and JVOL2(if the channel adaptor 20 belongs to the second storage device 4B), tothe other storage device (it is the second storage device 4B if thechannel adaptor 20 belongs to the first storage device 4A, and it is thethird storage device 4C if the channel adaptor 20 belongs to the secondstorage device 4B) (SP50).

The channel adaptor 20 determines whether a resynchronization of suchcopy pair is possible or not, based on a response or not from the otherstorage device or a content of the response or the like (SP51). Ifobtaining a negative result from the determination, the channel adaptor20 reports a failure of resynchronization to the first or the secondhost computer 3A or 3B of an issuing source of the resynchronizationcommand (SP60), and then finishes the resynchronization process to goback to the command receiving process described in FIG. 29.

If obtaining an affirmative result from the determination at the stepSP51, the channel adaptor 20 issues the difference bit map transmittingrequest to the other storage device (SP52) and then determines whetherthe difference bit map (the secondary inter-P/I difference bit map 51Bor the secondary inter-I/R difference bit map 50B) is transmitted fromthe other storage device or not (SP53).

If obtaining a negative result from such determination, the channeladaptor 20 goes to the step SP60, and, herein, if obtaining anaffirmative result, it merges the difference bit map acquired at thattime into the primary inter-P/I difference bit map 51A or into theprimary inter-I/R difference bit map 50A held in the share memory 22 ofthe self-storage device (SP54).

Subsequently, the channel adaptor 20 determines whether a site where theself-storage device is installed is an intermediate site (the secondsite 2B) or not (SP55).

A negative result from this determination means that the channel adaptor20 belongs to the first storage device 4A. At this time, the channeladaptor 20 goes to the step SP58.

On the contrary, an affirmative result from the determination at thestep SP55 means that the channel adaptor 20 belongs to the secondstorage device 4B. At this time, the channel adaptor 20 merges theprimary inter-I/R difference bit map 50A stored in the share memory 22into the secondary inter-P/I difference bit map 51B (SP56), and thendestroys the primary inter-I/R difference bit map 50A (SP57)

Thereafter, the channel adaptor 20 changes a pair state stored in thepair state column 31B of an entry corresponding to the copy pair as anobject at that time, of the entries of the pair information table 31(FIG. 4) stored in the share memory 22, into “synchronization.” (SP58).

The channel adaptor 20 reports completion of resynchronization to thefirst host computer 3A or the second host computer 3B of the issuingsource of the resynchronization command (SP59), and then finishes theresynchronization process to go back to the command receiving processdescribed in FIG. 29.

(4-6) Pair State Checking Process in the Second Storage Device 4B

Meanwhile, a channel adaptor 20 connected to the third storage device 4Cof the channel adaptors 20 of the second storage device 4B checks a pairstate of the copy pair configured of the first and the second journalvolumes JVOL1 and JVOL2 by periodically executing the pair statemanaging process shown in FIG. 34, with the command receiving processdescribed above in FIGS. 29 to 33.

Such channel adaptor 20 begins the pair state managing process shown inFIG. 34 periodically. The channel adaptor 20 determines whether or notthe pair state of the copy pair configured of the first and the secondjournal volumes JVOL1 and JOVL2 is “synchronization” (whether or not thejournal stored in the first journal volume JVOL1 is normally transmittedto the second journal volume JVOL2), based on the pair information table31 (FIG. 4) stored in the share memory 22 (SP70).

If obtaining a negative result from this determination, the channeladaptor 20 finishes such pair state managing process. On the contrary,if obtaining an affirmative result from this determination, channeladaptor 20 checks whether an obstacle is occurring in the network 6Bbetween the third storage device 4C and itself (SP71). In detail, thechannel adaptor 20 transmits a signal for check to the third storagedevice 4C and then checks a response or not for the signal for check.

The channel adaptor 20 determines whether an obstacle is occurring ornot in the network 6B, based on a check result at the step SP71 (SP72).

If obtaining a negative result from this determination, the channeladaptor 20 stores an invalid value (0) as a path disconnection detectingtime, in the path disconnection detecting time column 32E of an entrycorresponding to the journal group JG1 to which the first journal volumeJVOL1 belongs, of the entries of the journal group setting informationtable 32 (FIG. 5) (SP73), and then finishes the pair state managingprocess.

On the contrary, if obtaining an affirmative result from thedetermination at the step SP72, the channel adaptor 20 determineswhether the path disconnection detecting time set in the pathdisconnection detecting time column 32E is an invalid value (0) or not(SP74). In this case, if obtaining an affirmative result from thisdetermination, the channel adaptor 20 stores a current time in the pathdisconnection detecting time column 32E (SP75), and then goes to thestep SP66.

In contrast, if obtaining a negative result from the determination atthe step SP74, the channel adaptor 20 reads a path obstacle detectionchecking time stored in the path obstacle detection checking time column32D of an entry corresponding to the journal group JG1 to which thefirst journal volume JVOL1 belongs, of the entries of the journal groupsetting information table 32, and then determines whether or not thecorresponding path obstacle detection checking time elapsed fromtransmitting of a signal for check to the third storage device 4C in thestep SP71 (SP76).

If obtaining a negative result from this determination, the channeladaptor 20 finishes such pair state managing process. In contrast, ifobtaining an affirmative result from this determination, the channeladaptor 20 performs the synchronization unlocking process describedabove in FIG. 31 (SP77), and then finishes such pair state managingprocess.

(4-7) Pair State Managing Process in the Third Storage Device 4C

On the other hand, FIG. 35 shows a process content of the read journalprocess performed by a channel adaptor 20 connected to the secondstorage device 4B of the channel adaptors 20 of the third storage device4C. Such channel adaptor 20 performs the read journal process shown inFIG. 35 at the time of issuing the read journal command to the secondstorage device 4B, carrying out the command receiving process describedabove in FIGS. 29 to 33.

In other words, such channel adaptor 20 begins the read journal processat the time of issuing the read journal command to the second storagedevice 4B. The channel adaptor 20, referring to the pair informationtable 31 (FIG. 4) stored in the share memory 22, determines whether ornot a pair state of the copy pair configured of the first and the secondjournal volumes JVOL1 and JVOL2 is “synchronization” (whether or not thejournal stored in the first journal volume JVOL1 is normally transmittedto the second journal volume JVOL2) (SP80).

If obtaining a negative result from this determination, the channeladaptor 20 finishes such read journal process. In contrast, if obtainingan affirmative result from this determination, the channel adaptor 20transmits the read journal command to the second storage device 4B(SP81).

Thereafter, the channel adaptor 20 determines whether an obstacle isoccurring in the network 6B connecting the second storage device 4B andthe third storage device 4C, based on a response or not from the secondstorage device 4B for the read journal command (SP82).

If obtaining a negative result from this determination (that is, if ajournal is transmitted from the second storage device 4B in response tosuch read journal command), the channel adaptor 20 stores an invalidvalue (0) as a path disconnection detecting time, in the pathdisconnection detecting time column 32E of an entry corresponding to thejournal group JG2 to which the second journal volume JVOL2 belongs, ofthe entries of the journal group setting information table 32 (FIG. 5)(SP83). Successively, the channel adaptor 20 stores the journal acquiredat that time in the cache memory 23 and then finishes such read journalprocess.

In contrast, if obtaining an affirmative result from the determinationat the step SP82, the channel adaptor 20 processes the steps SP85 toSP88 in the same manner as the steps SP74 to SP77 of the pair statemanaging process described above in FIG. 34, and then finishes such readjournal process.

(5) Effect of the Present Embodiments

As above, in the storage system 1 according to the present embodiments,since, upon stopping of the remote copy among the first to the thirdstorage devices 4A to 4C, the respective second and the third storagedevices 4B and 4C manage locations in the third logical volume VOL3where the data held thereby are to be written, and, upon transmitting ofthe data again, the respective second and the third storage devices 4Band 4C aggregate the locations in the third logical volume VOL3 managedthereby, the data to be written in the respective aggregated locationsin the third logical volume VOL3 being transmitted from the firststorage device 4A to the third storage device 4C via the second storagedevice 4B, just a difference between the first logical volume VOL1 andthe third logical volume VOL3 is copied at the time of transmitting ofthe data again among the first to the third storage devices 4A to 4C.

Therefore, the present storage system 1 can recover a remote copy into anormal state at a very short time, compared with a conventional methodof copying all of the data in the first logical volume VOL1 to the thirdlogical volume VOL3.

(6) Other Embodiments

Although, in the above-described embodiments, a case that the presentinvention is applied to the storage system 1 having the second site 2Bof only one has been described, the present invention is not limitedthereto; the present invention is also applicable to a storage system inwhich the second sites 2B are installed in plural and the second storagedevices 4B installed in the respective second sites 2B are connected ina cascade.

In addition, although a case that a logical volume of the second storagedevice 4B set as a copy pair of the first logical volume VOL1 of thefirst storage device 4A is the virtual volume VVOL has been described inthe above-described embodiments, the present invention is not limitedthereto; a logical volume VOL1 with substance may be adopted instead ofsuch virtual volume VVOL.

1. A storage system comprising: a first storage device, configured to be installed in a first site, including a first adapter and a first cache memory and providing a primary logical volume in which data received from a host computer via the first adapter and the first cache memory is written; a second storage device, configured to be installed in a second site, including a second adapter and a second cache memory and providing a virtual logical volume, which has no structure for storing data received from the first storage system via the second adapter and the second cache memory; and a third storage device, configured to be installed in a third site, including a third adapter and a third cache memory and providing a secondary logical volume in which data received from the second storage system via the third adapter and the third cache memory is written, wherein data written in the primary logical volume of the first storage device are remote-copied to the secondary logical volume of the third storage device via the virtual logical volume of the second storage device, wherein, when the storage system stops the remote-copy between the second storage device and the third storage device, the first storage device does not transmit data to be remote-copied to the virtual logical volume of the second storage device, and each of the first storage device, the second storage device and the third storage device manages an address location of each of one or more data, each of the one or more data being held by the respective storage device as data to be remote-copied/written to the secondary logical volume of the third storage device, and wherein, when the storage system resumes the remote-copy from the primary logical volume of the first storage device to the secondary logical volume of the third storage device via the virtual logical volume of the second storage device, a plurality of the address locations managed by the first storage device, the second storage device, and the third storage device are aggregated, and data corresponding to each of the aggregated address locations is transmitted from the primary logical volume of the first storage device to the secondary logical volume of the third storage device via the virtual logical volume of the second storage device.
 2. The storage system according to claim 1, wherein the storage system stops the remote-copy between the second storage device and the third storage device based on an obstacle in a network between the second storage device and the third storage device.
 3. The storage system according to claim. 1, wherein each of the first storage device, the second storage device, and the third storage device manages the address location of each of one or more data by using a bitmap.
 4. The storage system according to claim 1, wherein the address location managed by the first storage device and an aggregated address location between the second storage device and the third storage device are aggregated in the first storage device after a plurality of address locations managed by the second storage device and the third storage device are aggregated in the second storage device.
 5. The storage system according to claim 1, wherein data written in the primary logical volume of the first storage device are remote-copied to the secondary logical volume of the third storage device via the virtual logical volume of the second storage device and a journal volume of the second storage device.
 6. The storage system according to claim 1, wherein the second storage device manages the address location by using a journal volume of the second storage device.
 7. A remote copy recovery method in a storage system comprising: a first storage device, configured to be installed in a first site, including a first adapter and a first cache memory and providing a primary logical volume in which data received from a host computer via the first adapter and the first cache memory is written; a second storage device, configured to be installed in a second site, including a second adapter and a second cache memory and providing a virtual logical volume, which has no structure for storing data received from the first storage system via the second adapter and the second cache memory; and a third storage device, configured to be installed in a third site, including a third adapter and a third cache memory and providing a secondary logical volume in which data received from the second storage system via the third adapter and the third cache memory is written, wherein data written in the primary logical volume of the first storage device are remote-copied to the secondary logical volume of the third storage device via the virtual logical volume of the second storage device, the method comprising the steps of: managing, by each of the first storage device, the second storage device and the third storage device, an address location of each of one or more data, each of the one or more data being held by the respective storage device as data to be remote-copied/written in the secondary logical volume of the third storage device, when the storage system stops the remote-copy between the second storage device and the third storage device and the first storage device does not transmit data to be remote-copied to the virtual logical volume of the second storage device; and aggregating a plurality of the address locations managed by the first storage device, the second storage device, and the third storage device and transmitting data corresponding to each of the aggregated address locations from the primary logical volume of the first storage device to the secondary logical volume of the third storage device via the virtual logical volume of the second storage device, when the storage system resumes the remote-copy from the primary logical volume of the first storage device to the secondary logical volume of the third storage device via the virtual logical volume of the second storage device.
 8. The remote copy recovery method according to claim 7, wherein, in the step of managing an address location of each of one or more data, each of the first storage device, the second storage device, and the third storage device manages the address location of each of one or more data by using a bitmap.
 9. The remote copy recovery method according to claim 7, wherein, in the step of managing an address location of each of one or more data, the address location managed by the first storage device and an aggregated address location between the second storage device and the third storage device are aggregated in the first storage device after a plurality of address locations managed by the second storage device and the third storage device are aggregated in the second storage device.
 10. A first storage device which includes a first adapter and a first cache memory and is configured to be installed in a first site and to provide a primary logical volume in which data received from a host computer via the first adapter and the first cache memory is written, wherein data written in the primary logical volume of the first storage device are remote-copied to a secondary logical volume of a third storage device via a virtual logical volume of a second storage device, wherein, when remote-copy between the second storage device and the third storage device is stopped, the first storage device is configured to not transmit data to be remote-copied to the virtual logical volume of the second storage device and to manage an address location of one or more data being held by the first storage device as data to be remote-copied/written to the secondary logical volume of the third storage device, and wherein, when remote-copy from the primary logical volume of the first storage device to the secondary logical volume of the third storage device via the virtual logical volume of the second storage device is resumed, a plurality of the address locations managed by the first storage device are aggregated and data corresponding to the aggregated address locations is transmitted from the primary logical volume of the first storage device to the secondary logical volume of the third storage device via the virtual logical volume of the second storage device.
 11. The storage device according to claim 10, wherein the address location managed by the first storage device and an aggregated address location between the second storage device and the third storage device are aggregated in the first storage device after a plurality of address locations managed by the second storage device and the third storage device are aggregated in the second storage device. 